Image processing apparatus and control method

ABSTRACT

A control method includes executing, in a case that a change instruction is accepted, at least a process for displaying a first post-change layout image on a display unit with a priority over a second post-change layout image with a higher degree of similarity to the pre-change layout image than the first post-change layout image or a process for displaying the first post-change layout image on the display unit in a more emphasized manner than the second post-change layout image.

BACKGROUND Field of the Disclosure

The present disclosure generally relates to image processing and, moreparticularly, to an image processing apparatus and a control method.

Description of the Related Art

There is known a process for generating a layout image by arrangingimages represented by image data acquired by taking pictures with adigital camera or the like on a template. There is also known a layoutimage change process by which the generated layout image is changed inresponse to user instructions to display a new layout image. JapanesePatent No. 5449460 describes that a user is prompted to select any of aplurality of templates displayed on a screen. It also describes that atemplate used for a layout image generated by automatic layout ischanged to the template selected by the user and a new layout image isdisplayed.

However, according to the technique described in Japanese Patent No.5449460, the user is caused to select any of the plurality of templatesand then a post-change layout image is displayed. Accordingly, the userneeds to specify manually the post-change layout image and a desired(highly satisfactory) template to be used for the post-change layoutimage. That is, the technique described in Japanese Patent No. 5449460has an issue that it is difficult for the user to specify a highlysatisfactory layout image as post-change layout image and a highlysatisfactory template as template to be used for the post-change layoutimage.

SUMMARY

In light of the foregoing issue, the present disclosure generally makesit easy for the user to specify a highly satisfactory layout image aspost-change layout image and a highly satisfactory template as templateto be used for the post-change layout image.

According to one or more aspects of the present disclosure, a controlmethod is characterized, among other features, in causing a computer inan image processing apparatus to execute:

displaying on a display unit a pre-change layout image in which anarrangement image is arranged in a template;

accepting a change instruction for displaying a post-change layout imagedifferent at least partly from the pre-change layout image on thedisplay unit; and

executing, in a case that the change instruction is accepted, at least aprocess for displaying a first post-change layout image on the displayunit with a priority over a second post-change layout image with ahigher degree of similarity to the pre-change layout image than thefirst post-change layout image or a process for displaying the firstpost-change layout image on the display unit in a more emphasized mannerthan the second post-change layout image.

In addition, a control method according to the present disclosure may becharacterized in causing a computer in an image processing apparatus toexecute:

displaying on a display unit a pre-change layout image in which anarrangement image is arranged in a template;

accepting a change instruction for displaying a post-change layout imagedifferent at least partly from the pre-change layout image on thedisplay unit; and

executing, in a case that the change instruction is accepted, at least aprocess for displaying a first post-change template on the display unitwith a priority over a second post-change template with a higher degreeof similarity to a template used for the pre-change layout image thanthe first post-change template or a process for displaying the firstpost-change template on the display unit in a more emphasized mannerthan the second post-change template.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a hardware configuration of an imageprocessing apparatus.

FIG. 2 is a software block diagram of an album creation application.

FIG. 3 is a diagram illustrating a display screen provided by the albumcreation application.

FIG. 4 is a flowchart of an automatic layout process.

FIG. 5 is a diagram illustrating a table for management of imageanalysis information of image data.

FIGS. 6A to 6C are diagrams for describing segmentation of image datagroups.

FIG. 7 is a diagram for describing classification of scenes.

FIGS. 8A and 8B are diagrams for describing scoring of a main slot and asub slot.

FIGS. 9A to 9I are diagrams for describing selection of image data.

FIG. 10 is a diagram illustrating template groups for use in generationof a layout image.

FIG. 11 is a flowchart of a template change process.

FIG. 12 is a diagram illustrating an editing screen provided by thealbum creation application.

FIGS. 13A and 13B are diagrams that illustrate tables showing valuesrelated to the degree of similarities between templates.

FIG. 14 is a diagram illustrating a configuration of a print system.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the presentdisclosure will be described below in detail with reference to theaccompanying drawings. The following embodiments are not intended tolimit the present disclosure according to the claims. In addition, allthe combinations of characteristics described in relation to theembodiments are not necessarily essential to the solution of the presentdisclosure. Identical constituent elements will be given identicalreference numbers and descriptions thereof will be omitted.

In the following embodiments, an application program for creation of analbum (photo book) (hereinafter, called “album creation application”) isoperated on an image processing apparatus to generate an automaticlayout as described below. Images described below include, unlessotherwise specified, still images, moving images, frame images in movingimages, and still images, moving images, and frame images in movingimages in social networking service (SNS) servers.

First Embodiment

In the present embodiment, an album creation application is operated onan image processing apparatus to generate a layout by an automaticlayout function as described below.

An example of a print system according to the present embodiment will bedescribed with reference to FIG. 14. The print system includes an imageprocessing apparatus 100, an image forming apparatus 200, a network 300,an external server 400, and an image forming apparatus 500.

The image forming apparatus 200 executes an image forming process (printprocess) to form an image on a print medium by a recording materialbased on a print job received from the image processing apparatus 100 orthe like. In the present embodiment, the image processing apparatus 100transmits (outputs) generated layout information to an external server.However, for example, the image processing apparatus 100 may beconfigured to transmit the generated layout information as a print jobto the image forming apparatus 200. In this case, the image formingapparatus 200 creates the album based on the layout information.

The network 300 connects to the image processing apparatus 100 and theexternal server 400 and constitutes a communication network fortransmission of information between the two. The network 300 may be awired network or a wireless network.

The external server 400 accepts layout information described later fromthe image processing apparatus 100 via the network 300. That is, theexternal server 400 is a server that accepts orders of albums andmanages the albums. When the user of the image processing apparatus 100performs an album purchase procedure, the external server 400 causes theimage forming apparatus 500 to create an album by an image formingprocess based on the accepted layout information. After that, the albumcreated by the image forming apparatus 500 is delivered to the user whohas performed the album purchase procedure.

FIG. 1 is a block diagram of a hardware configuration of an imageprocessing apparatus. Examples of the image processing apparatus mayinclude personal computer (PC), smartphone, tablet terminal, camera,printer, or the like. In the present embodiment, the image processingapparatus is a PC. A central processing unit (CPU/processor) 101, whichmay include one or more processors, one or more memories, circuitry, ora combination thereof, may perform centralized control of the imageprocessing apparatus 100, and may implement the operations in thepresent embodiment by reading programs from a ROM 102 into a RAM 103 andexecuting the same, for example. FIG. 1 illustrates one CPU but aplurality of CPUs may be provided. The ROM 102 is a general-purpose ROMthat stores the programs to be executed by the CPU 101, for example. TheRAM 103 is a general-purpose RAM that is used as a working memory forstoring temporally various kinds of information at the time of executionof the programs by the CPU 101, for example. The hard disc drive (HDD)104 is a storage medium (storage unit) for storing a database holdingimage files and results of processing such as image analysis andtemplates to be used by the album creation application.

A display device 105 displays to the user the results of layout of userinterfaces (UIs) and images in the present embodiment. A keyboard 106and a pointing device 107 accept instructions from the user. The displaydevice 105 may include a touch sensor function. The keyboard 106 is usedby the user to input the number of double-page spreads of an album to becreated on the UI displayed on the display device 105, for example. Thepointing device 107 is used by the user to click a button on the UIdisplayed on the display device 105, for example.

A data communication unit 108 conducts communications with externaldevices via wired or wireless networks. The data communication unit 108transmits layout data generated by the automatic layout function toprinters and servers communicable with the image processing apparatus100, for example. A data bus 109 connects between the blocks illustratedin FIG. 1 in a mutually communicable manner.

The album creation application in the present embodiment is saved in anHDD 104 and is activated by the user double-clicking the icon for theapplication on the display device 105 by the pointing device 107 asdescribed later. In addition, the album creation application isinstalled from an external server via the data communication unit 108and saved in the HDD 104, for example.

<Automatic Album Layout>

FIG. 2 is a software block diagram of the album creation application.FIG. 2 illustrates software blocks relating to an automatic layoutprocessing unit that executes the automatic layout function. Theautomatic layout function is a function to lay out image data classifiedand selected from taken still images and moving images based on theircontents and attributes in a prepared template and display the same onthe display device 105.

An album creation condition designation unit 201 accepts designation ofalbum creation conditions according to an UI operation described laterby the pointing device 107 and outputs the same to the automatic layoutprocessing unit, for example. The designated conditions include the IDsfor image data and main character to be processed, the number ofdouble-page spreads of the album, template information, ON/OFF state forimage correction, ON/OFF state for the use of moving images, designationof an album mode, and others, for example. The double-page spread refersto a pair of adjacent pages that is printed on different sheets (orpages). In the album creation application in the present embodiment, alayout of one double-page spread is created in one display window. Thealbum creation condition designation unit 201 displays a setting screenas illustrated in FIG. 3, for example, and accepts an input into thescreen to accept designation of album creation conditions.

An image acquisition unit 202 acquires an image group (image data group)designated by the album creation condition designation unit 201 from astorage area such as the HDD 104. The image acquisition unit 202 mayacquire the image group from a storage area such as a server or an SNSserver on networks via the data communication unit 108. The image grouphere refers to candidates for image data for use in creation of analbum. For example, the album creation condition designation unit 201may designate January 1, XX to December 31, XX, as a condition regardingthe date and time when the image data as a layout target was generated(the pictures corresponding to the image data were taken) (hereinafter,called shooting date and time), for example. In this case, the imageacquisition unit 202 acquires all image data generated on January 1, XXto December 31, XX, as an image group.

The image data stored in the storage area may include, for example,still image data and clipped image data acquired by clipping frames outof moving images. The still image data and the clipped image data areacquired from imaging devices. The imaging devices may be included inthe image processing apparatus 100 or may be included in externaldevices (PC, smart phone, digital camera, tablet terminal, or the like)as devices outside the image processing apparatus 100. In the case ofacquiring image data from an external device, the image processingapparatus 100 acquires the image data via the data communication unit108. The image processing apparatus 100 may acquire the still image dataand the clipped image data from a network or a server via the datacommunication unit 108. The CPU 101 analyzes data accompanying the imagedata to determine where the individual image data was acquired.

An image conversion unit 203 converts pixel count information and colorinformation of the image data acquired by the image acquisition unit202. The pixel count information and color information to be convertedinto the image data by the image conversion unit 203 are decided inadvance, and the information is saved in the album creation applicationor a parameter file to be used in the album creation application. In thepresent embodiment, the image data acquired by the image acquisitionunit 202 is converted into image data in which the pixel count is 420 ona short side and the color information is sRGB.

An image analysis unit 204 analyzes image data. In the presentembodiment, the image analysis unit 204 analyzes the image data alreadyconverted by the image conversion unit 203. Specifically, the imageanalysis unit 204 acquires feature amounts from the converted imagedata, and executes object detection, face detection, recognition offacial expressions from detected faces, personal recognition of thedetected faces in the converted image data. Further, the image analysisunit 204 acquires shooting date/time information from data (for example,Exif information) accompanying the image data before conversion acquiredby the image acquisition unit 202. The shooting date/time information isnot limited to the Exif information but may be information on the dateand time when the image data was generated or updated. Alternatively,the shooting date/time information may be information on the date andtime when the image data was uploaded to a local server or an SNS serveror the image data was downloaded from a local server or an SNS server.The foregoing date/time information is also treated as shootingdate/time information in the following description. The local server isa storage area included in the image processing apparatus 100 such asthe HDD 104.

An image classification unit 205 executes scene segmentation and sceneclassification described later on an image data group by using objectdetection result information such as the shooting date/time information,the number of images, and the detected face information. The scenesinclude shooting scenes such as “travel”, “daily life”, and “weddingceremony”. Each of the scenes can also be called a collection of imagedata generated on one shooting occasion, for example.

An image scoring unit 207 gives a score to each of the image data suchthat the image data suitable for a layout receive high scores. The imagescoring unit 207 conducts scoring according to the information obtainedby the image analysis unit 204 and the information obtained by the imageclassification unit 205. Other information may be used additionally oralternatively. In the present embodiment, the image scoring unit 207gives a score to each of the image data such that the image dataincluding a main character ID input from a main character informationunit receive high scores.

A user information input unit 206 inputs ID (identification information)of a main character designated by the album creation conditiondesignation unit 201 into the image scoring unit 207. The image scoringunit 207 is configured to add higher scores to the image data includingthe main character ID input by the user information input unit 206. Theuser information input unit 206 also inputs a priority mode designatedby the album creation condition designation unit 201 into the imagescoring unit 207. The image scoring unit 207 is configured to add higherscores to the image data including one or more objects input by the userinformation input unit 206.

A double-page spread number input unit 208 inputs the number ofdouble-page spreads of an album designated by the album creationcondition designation unit 201 into a double-page spread allocating unit209.

The double-page spread allocating unit 209 segments an image group(grouping) and allocates the segmented parts to the double-page spreads.The double-page spread allocating unit 209 segments the image groupaccording to the input number of double-page spreads and allocates partsof the image group to the double-page spreads.

An image selection unit 210 selects image data representing arrangementimages to be arranged in a template from the image group allocated tothe double-page spreads by the double-page spread allocating unit 209,based on the scores given by the image scoring unit 207.

A template input unit 211 reads a plurality of templates according tothe template information designated by the album creation conditiondesignation unit 201 from the HDD 104 and inputs the templates into animage layout unit 212. In the present embodiment, the plurality oftemplates is saved in the album creation application in the HDD 104. Theplurality of templates includes, for example, information on the sizesof the entire templates and information on the numbers, sizes, andpositions of slots included in the templates.

An image layout unit 212 decides the layout of a double-page spread.Specifically, the image layout unit 212 selects a template suitable forthe image data selected by the image selection unit 210 from theplurality of templates input by the template input unit 211, and decidesthe arrangement positions of the images. Accordingly, the layout of thedouble-page spread is decided.

A layout information output unit 215 outputs layout information fordisplaying the layout image on the display device 105 according to thelayout decided by the image layout unit 212. The layout image is, forexample, an image in which the arrangement images represented by theimage data selected by the image selection unit 210 are arranged in theselected template. The layout information is bitmap data indicating theimages.

An image correction unit 214 executes correction processing such asdodging correction (brightness correction), red-eye correction, andcontrast correction. A correction condition input unit 213 inputs ON/OFFstate of image correction designated by the album creation conditiondesignation unit 201 into the image correction unit 214.

The image data output from the layout information output unit 215 isdisplayed on the display device 105 in a format as illustrated in FIG.3, for example.

When the album creation application according to the present embodimentis installed into the image processing apparatus 100, the OS running onthe image processing apparatus 100 generates a starting icon on a topscreen (desktop) displayed on the display device 105. When the userdouble-clicks on the start icon with the pointing device 107, theprogram for the album creation application saved in the HDD 104 isloaded into the RAM 103. The program loaded into the RAM 103 is executedby the CPU 101 to start the album creation application. The program maybe stored in the ROM 102.

FIG. 3 is a diagram illustrating an example of a UI configuration screen301 provided by the started album creation application. The UIconfiguration screen 301 is displayed on the display device 105. Whenthe user sets album creation conditions described later via the UIconfiguration screen 301, the album creation condition designation unit201 acquires the setting contents designated by the user. A path box 302on the UI configuration screen 301 indicates the saving place (path) ofan image/moving image group to be a target of album creation in the HDD104. A folder selection button 303 is clicked by the user with thepointing device 107 to display folders including the image/moving imagegroup to be a target of album creation in a tree structure in auser-selectable manner. Then, the folder path including the image/movingimage group selected by the user is displayed in the path box 302.

Main character designating icons 304 are icons for the user to designatethe main character, and have persons' facial images displayed as icons.The person corresponding to the icon selected by the user is set as themain character of the album to be created. The main characterdesignating icons 304 are also used to specify the main character as acentral figure from persons seen in the images represented by the imagedata to be analyzed. The main character designating icons 304 are thefacial images of persons selected by the user or decided by a methoddescribed later from the facial images of persons registered with a facedatabase, for example. The main character may be automatically setaccording to the procedure illustrated in FIG. 4.

A double-page spread number box 305 accepts the setting for the numberof double-page spreads of the album from the user. The user inputs afigure via the keyboard 106 directly into the double-page spread numberbox 305 or inputs a figure from a list with the pointing device 107 intothe double-page spread number box.

Template designating icons 306 are illustration images according to thetastes of templates (pop, chic, and others). The template according tothe icon selected by the user is set as the template for use in thealbum to be created. In the present embodiment, each of the templateshas image arrangement frames (slots) for arrangement of the image data.The image data is embedded into the slots in the template to completeone layout image.

A mode designation unit 307 has icons corresponding to the modes of thealbum to be created. The mode of an album refers to the mode for layingout images including a predetermined object in a template on a prioritybasis. An album in each mode has more images of the object according tothe mode. In the present embodiment, there may be three modes including,for example, “person”, “animal”, and “food”. The mode of an album canalso be called the theme of the album, for example. When the “animal”mode is selected for an album, for example, the images including animalsare laid out in a template on a priority basis. In addition, there maybe a mode for laying out image data representing images of objects otherthan the foregoing three in the template on a priority basis. Aplurality of modes may be selected at the same time. In that case,images of one or more of the plurality of objects corresponding to theplurality of selected modes may be laid out in the template on apriority basis. Then, the mode corresponding to the selected icon is setas the mode of the album to be created.

The modes of the album are not limited to the foregoing three but may beany other modes such as “building”, “vehicle”, “flower”, etc., forexample.

A checkbox 308 accepts the ON/OFF setting of image correction from theuser. An OK button 309 is a button for accepting the confirmation ofcompletion of the setting from the user. When the user presses the OKbutton 309, the album creation condition designation unit 201 outputsthe setting information on the screen 301 to modules of the automaticlayout processing unit corresponding to the setting information.

A reset button 310 is a button for resetting the setting information onthe UI configuration screen 301.

Settings other than the foregoing settings may be made on the UIconfiguration screen 301. For example, the settings on moving images andthe settings on the destination of acquisition of image/moving imagedata may be made.

A server name box indicates the name of a server or an SNS including theimage group for use in the creation of the album. When a login to adesignated server or SNS is completed by the user via a login screen,the CPU 101 can acquire image data from the designated server or SNS.

A moving image use checkbox accepts from the user the setting on whetherthe moving image in the folder designated in the path box 302 or theserver or SNS designated in the server name box is to be used in thecreation of the album.

A target period box accepts from the user the setting on the conditionsfor the shooting date and time of the image group or the moving imagegroup as a target of the album creation.

FIG. 4 is a flowchart of a process by the automatic layout processingunit in the album creation application. The process in the flowchartillustrated in FIG. 4 is implemented by the CPU 101 reading a programfrom the HDD 104 into the ROM 102 or the RAM 103 and executing the same,for example. The automatic layout process will be described withreference to FIG. 4.

In S401, the CPU 101 subjects the image data acquired by the imageacquisition unit 202 to analysis by the image conversion unit 203 togenerate analyzed image data. Specifically, the CPU 101 converts by theimage conversion unit 203 the image data in the image data group in theHDD 104 designated by the album creation condition designation unit 201into the analyzed image data with a desired number of pixels and colorinformation. The image data acquired by the image acquisition unit 202is data corresponding to the still images or moving images acquired bythe CPU 101 from the folder, server, or SNS designated in the path box302 by the image acquisition unit 202.

In S402, the CPU 101 acquires feature amounts from the analyzed imagedata (the image data acquired and analyzed in S401) by the imageanalysis unit 204. For example, the CPU 101 acquires the shooting dateand time from the Exif information accompanying the image data acquiredfrom the HDD 104. The feature amount is focus, for example.

In S403, the CPU 101 executes face detection on the analyzed image datagenerated in S401 by the image analysis unit 204. The CPU 101 extracts afacial image and acquires the upper left coordinate value and lowerright coordinate value of the detected facial image. With these twokinds of coordinates, the CPU 101 can acquire the position of the facialimage and the size of the facial image. In this case, a strongdiscriminator may be generated by Adaboost to detect objects other thanfaces, including animals such as dog and cat, flowers, food, buildings,and stationary articles. Accordingly, the CPU 101 can detect objectsother than facial images.

In S404, the CPU 101 performs personal recognition by the image analysisunit 204. The CPU 101 determines the degree of similarity between eachof representative facial images and the extracted facial image, and setsthe ID of the representative facial image with the highest degree ofsimilarity equal to or greater than a threshold as the ID of thedetected facial image. When the degrees of similarity to all therepresentative facial images saved in a face dictionary database arelower than the threshold, the CPU 101 registers the extracted facialimage with a new personal ID in the face dictionary database.

In S405, the CPU 101 executes an object recognition process on theanalyzed image data generated in S401 by the image analysis unit 204 torecognize objects in the images represented by the image data. Theobject recognition process can be performed by a publicly known method.In the present embodiment, objects are recognized by a discriminatorgenerated by DeepLearning. The image analysis unit 204 recognizes objectimages to acquire the types of objects including pets such as dog andcat, flowers, food, buildings, and stationary articles.

The CPU 101 differentiates between the image analysis informationacquired in S402 to S405 by ID for identifying each of the images asillustrated in FIG. 5, and stores the same in a storage area such as theROM 102. For example, as illustrated in FIG. 5, the shooting date/timeinformation and focus determination results acquired in S402 and thenumber and position information of the facial images detected in S403,and the objects detected in S405 are stored in a table format. Theposition information of the facial images is stored in distinction fromone another by the personal ID acquired in S404.

In S406, the CPU 101 determines whether S401 to S405 are completed onall the image data in the image data group in the HDD 104 designated bythe album creation condition designation unit 201. When not determiningthat S401 and S405 are completed, the CPU 101 repeats S401 and thesubsequent steps. When determining that S401 and S405 are completed, theCPU 101 moves the process to S407.

In S407, the CPU 101 executes scene segmentation by the imageclassification unit 205. The scene segmentation refers to segmenting allthe images in the image group designated by the album creation conditiondesignation unit 201 for each of the scenes and managing the same as aplurality of image groups. In the following description, the imagegroups obtained by segmenting all the image data (main image group) willbe called sub image groups.

FIG. 6A illustrates an example of grouping of the captured image data.In FIGS. 6A to 6C, the lateral axis indicates the shooting date and time(older toward the left side and newer toward the right side), and thevertical axis indicates the number of the captured image data. In thecase of FIG. 6A, the captured image data group is segmented into eightsub image groups 601 to 608. In FIG. 6A, the arrows indicate theboundaries between the groups.

In S408, the CPU 101 executes scene classification by the imageclassification unit 205. Specifically, the CPU 101 gives scores to thesegmented image groups having undergone scene segmentation in S407 bythe type of scene and classifies the segmented image groups to the scenewith the highest score. In the following description, the scoring inS408 will be called scene classification and scoring. In the presentembodiment, the image classification unit 205 classifies the image datahaving undergone the scene segmentation to any of scenes “travel”,“daily life”, and “ceremony”, for example. Before the start of theautomatic layout processing unit as illustrated in FIG. 4, the usercollects and designates a plurality of image data judged as being in thescene of “travel”, “daily life”, or “ceremony” to generate a sceneclassification table. The scene classification table is a table storingthe information on the feature amounts corresponding to the types of thescenes, which is used for the scene classification and scoring.

In the present embodiment, a table 700 illustrated in FIG. 7 is used asthe scene classification table. The table 700 stores the average valuesand standard deviations of shooting period 702, shot image number 703,and shot person number 704 in association with scene IDs 701.

In S409, the CPU 101 determines whether the scene classification in S408is completed on all the segmented image data groups obtained in S407(here, the segmented image data groups corresponding to the segmentedscenes illustrated in FIG. 6A). When not determining that the sceneclassification is completed, the CPU 101 repeats S408 and the subsequentsteps on the new segmented image data groups yet to be processed. Whendetermining that the scene classification is completed, the CPU 101moves the process to S410.

In S410, the CPU 101 executes main character setting by the imagescoring unit 207. The main character setting is made on the image datagroup designated by the user by either of two automatic and manualsetting methods.

In S411, the CPU 101 executes priority mode setting by the image scoringunit 207. In the present embodiment, an album is created in any ofmodes. The modes for use in the album creation include a plurality ofmodes corresponding to the types of objects (for example, “person” mode,“animal” mode, and others). The priority mode setting is a process forsetting the mode to be used. The mode of the album may be set manuallyby the user or may be set automatically based on the characteristics ofthe image data acquired by the image acquisition unit 202.

In S412, the CPU 101 executes image scoring by the image scoring unit207. The image scoring means giving scores to the image data from aviewpoint of estimation described later, which is referred to at thetime of selection of image data for use in layout described later.

FIG. 10 is a diagram illustrating a template group for use in layout ofthe image data. The plurality of templates included in the templategroup corresponds to double-page spreads. A template 1001 is onetemplate. A main slot 1002 constitutes a main slot, and sub slots 1003and 1004 constitute sub slots. The main slot 1002 is the main slot(frame where images are to be laid out (arranged)) in the template 1001,which is larger in size than the sub slots 1003 and 1004. In onetemplate, image data classified into the same type of scene are laidout. In S410, the CPU 101 performs the image scoring process to giveboth a score for the main slot and a score for the sub slotscorresponding to the scene of the type of the image data.

In the image scoring, a slot feature table is used. The slot featuretable stores the information on the feature amounts of the images to beused in the main slot and the sub slots by the type of the scene. FIG.8A is a diagram illustrating the characteristics of the image data to beused in the main slot and the sub slots in the templates correspondingto the scenes of the types “travel”, “daily life”, and “ceremony”.Accordingly, scoring for both the main slot and the sub slots isexecuted on the image data.

In the present embodiment, the CPU 101 further adds scores to the scorescalculated as described above based on the mode designated by the albumcreation condition designation unit 201.

The CPU 101 performs image scoring on each of the image data in theimage data group designated by the user. The scores added by the imagescoring constitute a criteria for selection in an image selectionprocess in subsequent step S423. Accordingly, in the image selectionprocess described later, the CPU 101 selects image data representing theimages including objects in the category corresponding to the mode ofthe album set in S411 with a priority over the image data representingthe images not including the objects.

FIG. 8B illustrates an example of scoring results by the layout scoring.For example, an image ID1 is given a score of 20 for the main slot, andan image ID2 is given a score of 45 for the main slot. This means that,out of the images for the main slot, the image ID2 is closer to thecriteria for the user's judgment.

In S413, the CPU 101 determines whether the layout scoring by the imagescoring unit 207 is completed on all the image data in the image datagroup designated by the user. When not determining that the layoutscoring is completed, the CPU 101 repeats S413 and the subsequent stepson the subsequent unprocessed images. When determining that the layoutscoring is completed, the CPU 101 proceeds to S414.

In the present embodiment, the CPU 101 lays out the image data includedin the plurality of scenes, in the plurality of templates correspondingto the plurality of scenes (the plurality of double-page spreads) by theimage layout unit 212. Accordingly, it is necessary that there is amatch between the number of scenes and the number of templates (thepredetermined number of double-page spreads).

In S414, the CPU 101 determines by the double-page spread allocatingunit 209 whether the number of segments obtained by the scenesegmentation is the same as the number of double-page spreads of thealbum input by the double-page spread number input unit 208 in S407.When not determining that the numbers are the same, the CPU 101 proceedsto S415, and when determining that the numbers are the same, the CPU 101proceeds to S418.

In S415, the CPU 101 determines by the double-page spread allocatingunit 209 whether the number of segments obtained by the scenesegmentation in S406 is smaller than the number of double-page spreads(the number of templates for use in the album) input by the double-pagespread number input unit 208. When determining that the number ofsegments is not smaller (larger), the CPU 101 proceeds to S417, and whendetermining that the number of segments is smaller, the CPU 101 proceedsto S416. For example, when the number of scene segments is eight and thenumber input by the double-page spread number input unit 208 is ten asillustrated in FIG. 6A, the CPU 101 proceeds to S416.

In S416, the CPU 101 executes sub scene segmentation by the double-pagespread allocating unit 209. The sub scene segmentation meanssub-segmentation of the segmented scenes in the case where the number ofscene segments is smaller than the number of double-page spreads of thealbum. Descriptions will be given as to the case where the number ofscene segments is eight and the designated number of double-page spreadsof the album is ten as illustrated in FIG. 6A. FIG. 6B illustrates theresults of sub scene segmentation of the scenes illustrated in FIG. 6A.In this case, the segmentation intervals may be shortened so thatsegmentation is made at points shown by dashed arrows and thus thenumber of segments becomes ten.

The standard for segmentation will be described. Among the segmentsillustrated in FIG. 6A, segments with a large number of image data aresearched for. In this case, two points with a large number of image dataare determined to increase the number of segments by two from eight toten. That is, the scenes to be sub-segmented are designated in orderfrom the scene corresponding to the segmented image data group includingthe largest number of image data. For the scenes corresponding to thesegmented image data groups with the same number of image data, thescene with the larger maximum value of the difference in shooting dateand time between the image data included in the corresponding segmentedimage data group is selected. Still if the selection is not possible,the scene corresponding to the segmented image data group including theimage data generated earlier may be sub-segmented on a priority basis,for example.

Referring to FIG. 6A, the scenes with a large number of image dataincluded in the corresponding segmented image data group are scene 5,scene 1, and scene 2 in descending order. Scenes 1 and 2 have the samenumber of data, but scene 2 is set as segmentation target because scene2 has a larger time difference between the first-generated image dataand the last-generated image data in the corresponding segmented imagedata group. That is, in the case of FIG. 6A, scenes 5 and 2 aresegmented.

First, the segmentation of scene 2 will be described. Scene 2 has twopeaks of image data, and the image data included in the two peaks aredifferent in shooting date. Accordingly, scene 2 is segmented at a pointshown by a dashed arrow illustrated in FIG. 6B corresponding to the partwhere the shooting date is changed. Next, the segmentation of scene 5will be described. Scene 5 has three peaks of image data, and the imagedata included in the three peaks are different in shooting date as inthe case of scene 2. Scene 5 has two points where the shooting date ischanged. In this case, however, scene 5 is segmented by shooting datesuch that differences in the number of data after the segmentationbecome small. That is, scene 5 is segmented at a point shown by a dashedarrow illustrated in FIG. 6B. In such a manner as described above, thenumber of segments, that is, the number of scenes is changed from eightto ten. Specifically, the scene sub-segmentation is performed such that,when the selected scene includes image data different in shooting date,the image data groups different in shooting date are new scenes. Whenthe shooting extends across three or more days, scene segmentation isperformed such that the differences in the number of image data betweenthe segmented scenes become minimum and the images of the same shootingdate are included in the same scene. An example of segmentation at thepoint different in shooting date has been described so far. However,when the shooting date of the image data in the scene to be segmented isa single day, the segmentation is made at a point where the timedifference in the shooting hour in the single day becomes maximum. Inaddition, in this example, the segmentation is made at the point wherethe shooting date is changed. However, when there are points with alarge number of image data in a single day, the segmentation may be madeat a point with the maximum time difference in the single day. Accordingto the procedure described above, the number of scenes is made identicalto the number of double-page spreads. The scenes generated by thesegmentation may be anew classified or may take over the classificationbefore the segmentation.

In S417, the CPU 101 executes scene integration by the double-pagespread allocating unit 209. The scene integration means that, in thecase where the number of scene segments is larger than the number ofdouble-page spreads of the album, the segmented scenes are integrated.Descriptions will be given as to the case where the number of scenesegments is eight and the designated number of double-page spreads issix as illustrated in FIG. 6A as an example. FIG. 6C illustrates theresult of scene integration from FIG. 6A. The scenes before and afterthe points shown by broken lines are integrated to change the number ofsegments to six.

The criteria for integration will be described. First, the CPU 101detects scenes with a small number of image data from the segmentedscenes. In this example, two scenes with a small number of image dataare detected to decrease the number of scenes from eight to six.Referring to FIG. 6A, the scenes with a large number of scenes includedin the corresponding segmented image data group are scene 8, scene 3,and scene 7 in ascending order. Scene 3 and scene 7 are the same in thenumber of image data.

Next, the CPU 101 sets scene 8 with the smallest number of image dataincluded in the corresponding segmented image data group as a target ofintegration. Scene 3 and scene 7 are the same in the number of imagedata included in the corresponding segmented image data group, and thusthe CPU 101 selects either one of them as a target of integration. Inthis example, since scene 8 adjacent to scene 7 is a target ofintegration (scene 7 and scene 8 are to be integrated), the CPU 101 setsscene 3 as a target of integration.

Next, the CPU 101 decides whether the sub image data groups as a targetof integration are to be integrated into the scene earlier in theshooting date and time of the image data or later in the shooting dateand time of the image data included in the corresponding segmented imagedata group. At this time, the CPU 101 sets, out of the two scenesadjacent to the scene as a target of integration, the scene with asmaller time difference in the shooting date and time of the image dataincluded in the corresponding segmented image data group as the targetof integration. Accordingly, in the example of FIG. 6A, scene 2 or 4adjacent to scene 3 is set as the target of integration with scene 3. Incomparison between scene 3 and each of scenes 2 and 4 in the timedifferences in the shooting date and time between the images included inthe corresponding segmented image data group, the time differencebetween scene 3 and scene 4 is smaller. Accordingly, the CPU 101 decidesintegration of scene 3 with scene 4. That is, the integration is made atthe point shown by a broken line in FIG. 6C.

When there is only one adjacent scene as scene 8, the CPU 101 sets theone adjacent scene as the target of integration.

Accordingly, scene integration is made between the scenes before andafter the points shown by broken lines illustrated in FIG. 6C. The sceneintegration includes updating information indicating the image files inthe scene according to the scenes after the integration, for example.Specifically, the CPU 101 integrates the scenes such that there is amatch between the number of scenes and the number of double-pagespreads. Descriptions will be given as to the case where the number ofscene segments is eight and the designated number of double-page spreadsis six as illustrated in FIG. 6A as an example. FIG. 6C illustrates theresults of scene integration from FIG. 6A. The scenes before and afterthe points shown by broken lines are integrated to change the number ofsegments to six.

In S418, the CPU 101 executes double-page spread allocation by thedouble-page spread allocating unit 209. Through S414 to S417, the numberof scene segments and the number of designated double-page spreadsbecome the same. The CPU 101 allocates the segmented data groups fromthe beginning to the double-page spreads from the beginning in order ofshooting date and time. Specifically, the CPU 101 allocates the subimage groups to the double-page spreads of the album in the order ofshooting date and time. This makes it possible to create the album inwhich the sub image groups are aligned in the order of shooting date andtime.

In S419, the CPU 101 executes image selection by the image selectionunit 210. Descriptions will be given as to an example in which fourimage data are selected from the segments of the image data groupallocated to a double-page spread with reference to FIGS. 9A to 9I. Adouble-page spread is an area of two pages. However, each of the firstand last double-page spreads is an area of one page.

FIG. 9A illustrates the time difference in the shooting date and timebetween the image data at the earliest shooting date and time and theimage data at the latest shooting date and time out of the segmentedimage data group allocated to a double-page spread (divided shootingperiod), in other words, the shooting period of the segmented image datagroup. In this case, the image data are selected for the main slot andthe sub slot in this order. In this example, the template correspondingto the double-page spread includes the one main slot 1002. Accordingly,the image data selected at first is the image data for the main slot.From the image data corresponding to the divided shooting periodillustrated in FIG. 9B, the CPU 101 selects image data (1) with thehighest score for the main slot added in S416 as image data for the mainslot.

The second and subsequently selected image data are the image data forthe sub slot. The second and subsequent image data are selected by amethod as described below such that the image data do not concentrate onone part of the shooting period. First, the CPU 101 divides the dividedshooting period into two as illustrated in FIG. 9C. Next, as illustratedin FIG. 9D, the CPU 101 selects second image data from the image datagenerated during the divided shooting period in which the first imagedata is not selected (the period shown by a solid line in the drawing).As the second image data, the CPU 101 selects image data (2) with thehighest score for the sub slot from the image data generated during thedivided shooting period in which the first image data is not selected.Next, the CPU 101 further divides each of the divided shooting periodsillustrated in FIG. 9D into two as illustrated in FIG. 9E. Asillustrated in FIG. 9F, the CPU 101 selects third image data from theimage data generated during a divided shooting period in which none ofthe first image data and second image data is selected (the period shownby a solid line in the drawing). As the third image data, the CPU 101selects image data (3) with the highest score for the sub slot from theimage data generated during the divided shooting period in which none ofthe first image data and second image data is selected. As fourth imagedata, the CPU 101 selects the image data with the highest score for thesub slot from the image data generated during the divided shootingperiod in which none of the first to three image data is selected.

Next, descriptions will be given as to an example in which there is noimage generated during the divided shooting period in which none of thefirst to third image data is selected and no fourth image data can beselected from the image data generated during the divided shootingperiod. As illustrated in FIG. 9G, it is assumed that there is no imagedata generated during the divided shooting period (shaded in thedrawing) in which no image data is selected. In that case, the CPU 101further divides each of the divided shooting periods into two asillustrated in FIG. 9H. Next, as illustrated in FIG. 9I, the CPU 101selects the fourth image data from the images generated during a dividedshooting period in which no image data is yet selected other than thedivided shooting period in which it is recognized that there is no imagedata generated (the period shown by a solid line in the drawing). TheCPU 101 selects, as the fourth image data, image data (4) with thehighest score for the sub slot from the image data generated during thedivided shooting period.

FIG. 4 is referred to again. In S420, the CPU 101 decides image layoutby the image layout unit 212. Descriptions will be given as to anexample in which the contents of the designated template informationindicate that the number of the slots is three, and templates (a) to (p)illustrated in FIG. 10 corresponding to the template information areused.

First, the CPU 101 acquires the plurality of templates ((a) to (p)illustrated in FIG. 10) according to the template information designatedby the album creation condition designation unit 201. Next, the CPU 101decides the image layout for a double-page spread to be processed by theimage layout unit 212. At this time, the CPU 101 first decides thetemplate suitable for the image data selected in S419 from the pluralityof acquired templates.

In the present embodiment, the image data earlier in the shooting dateand time is laid out in the upper left slots and the image data later inthe shooting date and time is laid out in the lower right slots, of thetemplate. Accordingly, from the templates (a) to (p) illustrated in FIG.10, the CPU 101 first selects the templates in which the position of themain slot corresponds to the order of the shooting date and time of theimage data representing the main image. That is, for example, when theimage data representing the main image is in the third place in theorder of the shooting date and time, the CPU 101 selects the templatesin which the main slot is the third one of all the slots from the left.The image data are sorted in advance in the order of the shooting dateand time so that the CPU 101 can grasp the order of the shooing date andtime of the image data indicating the main image. In addition, from thethus selected templates, the CPU 101 finally selects the template inwhich the aspect ratio of the slot where the main image is to bearranged is identical (or close) to the aspect ratio of the main image.

The CPU 101 selects the template suitable for the sub image from thetemplates (a) to (p) illustrated in FIG. 10. Specifically, the CPU 101first selects the templates in which the position of the sub slotcorresponds to the order of the shooting date and time of the image dataindicating the sub image. In addition, from the thus selected templates,the CPU 101 finally selects the template in which the aspect ratio ofthe sub slot is identical (or close) to the aspect ratio of the subimage.

In this example, it is assumed that the shooting dates and times of theimages indicated by the three image data are in the order as illustratedin (a) of FIG. 10. It is also assumed that an image 1005 is an image forthe main slot (main image) and an image 1006 and an image 1007 areimages for the sub slots (sub images).

As described above, in the present embodiment, the image data earlier inthe shooting date and time is laid out in the upper left slots and theimage data later in the shooting date and time is laid out in the lowerright slots, of the template. The image 1005 as the main image is latestin the shooting date and time and is vertically oriented, and thus theCPU 101 selects templates (a) to (d) illustrated in FIG. 10.

The image 1006 as a sub image is earliest in the shooting date and timeand is vertically oriented. The image 1007 as a sub image is secondlatest in the shooting date and time and is horizontally oriented.Accordingly, the CPU 101 selects the template (b) illustrated in FIG.10. At this time, the CPU 101 specifies in which sub slots which subimages are to be arranged.

In this way, in S425, the CPU 101 decides the templates for use in thegeneration of the layout image and in which slots of the templates whichimages are to be laid out. Specifically, the CPU 101 manages theinformation on the slots included in the selected templates inassociation with the image data corresponding to the images to be laidout in the slots.

In S421, the CPU 101 executes image correction by the image correctionunit 214. As image correction, dodging correction (luminancecorrection), red-eye correction, and contrast correction are executed,for example.

In S422, the CPU 101 generates layout information by the layoutinformation output unit 215. Specifically, the CPU 101 manages the imagedata corresponding to the slots having undergone the image correction inS421, in association with the slots in the templates selected in S420.Then, the CPU 101 generates bit map data in which the images are laidout in the templates. At this time, the CPU 101 lays out the images in ascaled or non-scaled manner according to the size information of theslots.

In S423, the CPU 101 determines whether S419 to S422 are completed onall the double-page spreads. When not determining that S419 to S422 arecompleted, the CPU 101 repeats S419 and the subsequent steps. Whendetermining that S419 to S422 are completed, the CPU 101 terminates theautomatic layout process illustrated in FIG. 4.

After terminating the automatic layout process, the CPU 101 displays thelayout image in which the images are arranged in the templates on thedisplay device 105 based on the generated layout information. At thistime, the CPU 101 may display a plurality of layout images for creatingone album. In addition, the CPU 101 may transmit the generated layoutinformation to a printer such as the image forming apparatus 200 toprint the layout image. The layout image is printed to create the album.

In the foregoing automatic layout process, the templates and the imagedata are automatically selected in the album creation application(without accepting a selection instruction by the user) to generate thelayout image. However, the images represented by the layout informationdo not necessarily include the templates and the images represented bythe image data. This is because the layout information in the presentembodiment is used to create an album which includes some areas where noimages represented by image data are printed, called end paper, flyleaf, title page, and colophon, for example. In the present embodiment,the layout information also represents images corresponding to the endpaper, fly leaf, title page, and colophon. The data representing theseimages is not generated by the automatic layout process as describedabove. Accordingly, the data generated in advance for the imagescorresponding to the end paper, fly leaf, title page, and colophon isincluded in the layout information at some timing.

In the present embodiment, the details of the automatic layout processare not limited to the foregoing ones. For example, the method forselection of the templates for use in the album and the method forselection of the image data representing images to be arranged in thetemplates are not limited to the foregoing ones. At least, the layoutinformation is preferably generated without the user having to selectthe templates for use in the album or the image data representing theimages to be arranged in the templates.

<Editing of the Album>

After generating the layout information as described above, the CPU 101displays a screen for accepting the editing of the album represented bythe generated layout information. On this screen, the user can check thecontents of the album represented by the layout information generated bythe automatic layout process. Hereinafter, this screen will be calledediting screen. In the present embodiment, on the editing screen, one ofthe plurality of double-page spreads of the album represented by thegenerated layout information is displayed in an enlarged manner. Thedouble-page spreads to be displayed can be switched by the useroperation.

On the editing screen, the user can change the templates of the albumrepresented by the layout information generated by the automatic layoutprocess. A process for changing the templates and executing theautomatic layout process again will be called template change process.

FIG. 12 is a diagram illustrating an example of an editing screen 1201provided by the album creation application. The editing screen 1201 isdisplayed on the display device 105.

A double-page spread designation area 1202 is an area for accepting thedesignation of a double-page spread to be displayed in a layout resultdisplay area 1203 by the user's click operation with the pointing device107. In this case, the album to be created by automatic layout includeseight double-page spreads, and the double-page spread imagescorresponding to the double-page spreads are displayed in thedouble-page spread designation area 1202.

The layout result display area 1203 displays the layout resultcorresponding to the double-page spread (double-page spread as anediting target) designated by the user via the double-page spreaddesignation area 1202 in an enlarged manner. The layout result displayarea 1203 may display the album not by double-page spread but by page.There is no particular limit on the double-page spread to be displayedin the layout result display area 1203 displayed immediately after theautomatic layout process. For example, among the plurality ofdouble-page spreads, the first double-page spread (with the smallestdouble-page spread number) may be displayed or the double-page spread inthe highest importance level may be displayed immediately after theautomatic layout process. In the present embodiment, the double-pagespread to be displayed in the layout result display area 1203 displayedimmediately after the automatic layout process is the first one of thedouble-page spreads (the double-page spread with the double-page spreadname “front cover”).

A template change button 1204 is a button for changing the template usedfor the layout image corresponding to the double-page spread as anediting target and suggesting a new layout image to the user. The CPU101 accepts an instruction for changing the template by accepting thepress of the template change button 1204 by the user's click operationwith the pointing device 107. When the template change button 1204 ispressed, the template change process is executed to change the layoutimage for the double-page spread displayed in the layout result displayarea 1203. Specifically, the template used for the layout imagedisplayed at the time of the press of the template change button 1204 isreplaced by a different template to display a newly generated layoutimage. In the present embodiment, the images arranged in the slots areidentical between the pre-change layout image and the post-change layoutimage. The detailed procedure for changing the templates will bedescribed later. In the present embodiment, only the layout image forthe double-page spread displayed in the layout result display area 1203is changed by the press of the template change button 1204. However, thepresent disclosure is not limited to this embodiment. For example, aplurality of layout images may be changed in one batch.

An OK button 1205 is a button for specifying the layout image displayedin the layout result display area 1203 after the template changeprocess, as the layout image to be printed corresponding to thedouble-page spread specified as an editing target double-page spread.The user presses the OK button 1205 when he/she likes the layout imagenewly displayed by the template change process.

A reset button 1206 is a button for returning from the new layout imagedisplayed in the layout result display area 1203 after the templatechange process to the layout image having been displayed before thedisplay of the new layout image. In the present embodiment, when thereset button 1206 is pressed, the image to be displayed in the layoutresult display area 1203 is returned to the layout image to be printedcorresponding to the double-page spread specified as editing targetdouble-page spread.

A put-into-cart button 1207 is a button for ordering the album createdby the automatic layout. When the user executes a predetermined orderingoperation after the press of the put-into-cart button 1207, the layoutinformation set at that time is transmitted to the external server 400(uploaded) so that the album is created based on the layout information.Accordingly, the user edits arbitrarily the layout information for thealbum by the template change process and then presses the put-into-cartbutton 1207.

The issues of the present disclosure will be described. The userperforms the template change process when he/she does not like thetemplate used for the layout image generated by the automatic layoutprocess described in the flowchart of FIG. 4 or he/she wishes to seelayout images in other templates. At this time, in a mode in which aplurality of layout images in a plurality of other templates issuggested to the user who is prompted to select the post-change layoutimage, for example, the user has to search for the appropriate (highlysatisfactory) layout image by himself/herself. In addition, in somemode, each time an instruction for the template change process from theuser is accepted, one post-change layout image is suggested. In thismode, if the layout images suggested on a priority basis are possiblylow satisfactory for the user, the user also has to search for thehighly satisfactory layout image by himself/herself.

Accordingly, in the present embodiment, when an instruction for thetemplate change process from the user is accepted, layout images likelyto be highly satisfactory for the user are suggested to the user on apriority basis. Specifically, in the present embodiment, layout imagesin templates with a low degree of similarity to the template used forthe layout image as a target of the template change process aresuggested to the user on a priority basis.

The template change process will be described below in detail. FIG. 11is a flowchart of a process executed with the editing screen 1201displayed in the present embodiment. The flowchart illustrated in FIG.11 is implemented by the CPU 101 reading the program corresponding tothe album creation application from the HDD 104 into the ROM 102 or theRAM 103 and executing the same, for example.

In S1101, the CPU 101 accepts a user operation for designating adouble-page spread as an editing target from the double-page spreadsdisplayed in the double-page spread designation area 1202.

In S1102, the CPU 101 specifies the double-page spread selected by theaccepted operation as an editing target double-page spread, andspecifies the template used for the layout image corresponding to theediting target double-page spread (the pre-change layout image).

In S1103, the CPU 101 specifies the template (change candidate template)for use in the case where the template change process is to be performedon the layout image corresponding to the editing target double-pagespread. In the present embodiment, specifically, the CPU 101 specifiesthe template with a low degree of similarity in slot aspect ratio to thetemplate used for the layout image corresponding to the editing targetdouble-page spread, as change candidate template.

In the present embodiment, the template with a low degree of similarityin slot aspect ratio is specified from the template group including (a)to (p) illustrated in FIG. 10 as a template group with a high degree ofsimilarity in the number of slots. The slots in the templates (a) to (p)illustrated in FIG. 10 are assigned numbers in order from the left side.In the present embodiment, the degrees of similarity in slot aspectratio are calculated by comparison between the slots with the samenumber.

FIG. 13A is a table indicating the values relating to the degree ofsimilarity between the templates (a) to (p) illustrated in FIG. 10. Thistable provides the values relating to the degree of similarity betweenthe templates shown in the rows and the templates shown in the columns.As the value relating to the degree of similarity is smaller, the degreeof similarity between the two templates is higher (more similar).

A method for calculating the degree of similarity between the templateused for the layout image corresponding to the editing targetdouble-page spread and another template will be described. First, theelements as a target of comparison for the degree of similarity betweenthe templates (the slot aspect ratio in this example) are expressed invector. The distances between the expressed individual vectorsconstitute the values relating to the degree of similarity between thetemplates.

In this example, the aspect ratio of the slot with number n in atemplate (x) is expressed as ARx (n). In that case, the vector of theslot aspect ratio in the template with m slots is expressed as follows:{right arrow over (ARx)}=[ARx(n)ARx(n+1)ARx(n+2) . . . ARx(m)]  [Math.1]

In addition, a distance d between the vector indicated by the template(x) and the vector indicated by a template (y) is expressed as follows:

$\begin{matrix}{d = \sqrt{\begin{matrix}{{{{{ARx}(n)} - {{ARy}(n)}}}^{2} + {{{{ARx}\left( {n + 1} \right)} - {{ARy}\left( {n + 1} \right)}}}^{2} +} \\{{{{{{ARx}\left( {n + 2} \right)} - {{ARy}\left( {n + 2} \right)}}}^{2}\ldots} + {{{{ARx}(m)} - {{ARy}(m)}}}^{2}}\end{matrix}}} & \left\lbrack {{Math}\mspace{14mu} 2} \right\rbrack\end{matrix}$

For example, when a slot is a height of 4 and a width of 3, the aspectratio of the slot is 4/3. Accordingly, the vector indicated by atemplate (a) illustrated in FIG. 10 is expressed as follows:

$\begin{matrix}\left\lbrack \begin{matrix}\frac{4}{3} & \frac{4}{3} & \left. \frac{4}{3} \right\rbrack\end{matrix} \right. & \left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack\end{matrix}$

The vector indicated by a template (b) is expressed as follows:

$\begin{matrix}\left\lbrack \begin{matrix}\frac{4}{3} & \frac{4}{3} & \left. \frac{3}{4} \right\rbrack\end{matrix} \right. & \left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack\end{matrix}$

Further, the distance d between the vector indicated by the template (a)and the vector indicated by the template (b) is expressed by thefollowing equation:

$\begin{matrix}{d = {\sqrt{{{\frac{4}{3} - \frac{4}{3}}}^{2} + {{\frac{4}{3} - \frac{4}{3}}}^{2} + {{\frac{4}{3} - \frac{3}{4}}}^{2}} = 0.58}} & \left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack\end{matrix}$

In this way, the calculation results of the values relating to thedegree of similarity between the templates are shown in the tableillustrated in FIG. 13A. Referring to this table, the CPU 101 specifiesthe templates with a low degree of similarity to the template used forthe layout image corresponding to the editing target double-page spread(with a large value relating to the degree of similarity) as changecandidate templates. There is no particular limit on the number of thechange candidate templates, although there are four in the presentembodiment.

Specifically, when the template used for the pre-change layout image isthe template (a), for example, the row corresponding to the template (a)in the table are referred to. In addition, as illustrated in FIG. 13B,the values relating to the degree of similarity are in the columnsre-arranged in the ascending order. After the re-arrangement in thisway, the four templates with a low value relating to the degree ofsimilarity are specified as change candidate templates.

In the foregoing mode, the values relating to the degree of similarityof all the combinations in the template are determined. However, thepresent disclosure is not limited to this mode. For example, the valuesrelating to the degree of similarity between the template used for thelayout image corresponding to the editing target double-page spread andanother template may be determined. In addition, the album creationapplication may hold in advance the table illustrated in FIG. 13Ainstead of generating the table illustrated in FIG. 13A by calculation.In this case, the CPU 101 extracts the values corresponding to thetemplate used for the layout image corresponding to the editing targetdouble-page spread from the table held in advance, and specifies thechange candidate templates.

As described above, specified in the present embodiment are templatesidentical in elements other than the slot aspect ratio to those of thetemplate used for the layout image corresponding to the editing targetdouble-page spread but having a low degree of similarity in the slotaspect ratio to that of the layout image corresponding to the editingtarget double-page spread. That is, the templates with a low degree ofsimilarity in the slot aspect ratio are specified from the templategroup with a high degree of similarity in the elements other than theslot aspect ratio.

However, the present disclosure is not limited to this mode. Forexample, templates with a low degree of similarity in an element otherthan the slot aspect ratio (the number of slots, the area of the slots,the shape of the slots, the order of the slots, the coordinate positionsof the slots in the template, the number of overlaps between slots, andthe number of coupling portions of slots) may be specified. The numberof overlaps between slots refers to the number of overlapping portionsof slots in the template, and the number of coupling portions of slotsrefers to the number of portions not overlapping but coupling(contacting) of slots in the template. In addition, the templates with alow degree of similarity in not only one element but also two elementsmay be specified.

The user may be allowed to designate in what element the templates witha low degree of similarity is to be specified. Further, in the mode inwhich a plurality of elements can be designated, weighting may beexecutable in relation to the importance level of low degree ofsimilarity in each of the elements. In that case, the templates with alower degree of similarity in the element assigned a greater weight arespecified with a priority over the templates with a lower degree ofsimilarity in the element assigned a lower weight.

There is no limit on the template group from which the templates with alow degree of similarity are to be selected. In the present embodiment,the templates with a low degree of similarity are selected from thetemplate group with the same number of slots as that of the slots in thetemplate used for the layout image corresponding to the double-pagespread to be displayed. Alternatively, the templates with a low degreeof similarity may be selected from a different template group.

In S1104, the CPU 101 displays the layout image corresponding to theediting target double-page spread in the layout result display area1203.

In S1105, the CPU 101 determines whether the press of the templatechange button 1204 by the user is accepted. When making a YESdetermination, the CPU 101 proceeds to S1106, and when making a NOdetermination, the CPU 101 terminates the process in the flowchart.

In S1106, the CPU 101 executes the template change process.Specifically, the CPU 101 first specifies the template with the lowestdegree of similarity (the largest value relating to the degree ofsimilarity) from the change candidate templates not yet used in thetemplate change process. Then, the CPU 101 uses the specified template(post-change template) to perform S420 to S422 again to generate newlayout information. When all the change candidate templates aredisplayed due to repetition of the process, the CPU 101 makes a reset sothat all the change candidate templates are yet to be used and thenexecutes the process.

In S1107, the CPU 101 displays the layout image (post-change layoutimage) represented by the generated layout information in the layoutresult display area 1203. In the present embodiment, the images arrangedin the slots of the layout image displayed after the template changeprocess are identical to the images arranged in the slots of the layoutimage having been displayed before the template change process.

In S1108, the CPU 101 determines whether the press of the templatechange button 1204 by the user is accepted. When making a YESdetermination, the CPU 101 proceeds again to S1106, and when making a NOdetermination, the CPU 101 proceeds to S1109.

In S1109, the CPU 101 determines whether the press of the reset button1206 (reset operation) by the user is accepted. When making a YESdetermination, the CPU 101 proceeds again to S1110, and when making a NOdetermination, the CPU 101 proceeds to S1111.

In S1110, the CPU 101 returns (resets) from the new layout imagedisplayed in the layout result display area 1203 after the templatechange process to the layout image having been displayed before thedisplay of the new layout image. The layout image displayed by resettingmay be the layout image having been displayed immediately before thedisplay of the layout image after the reset button 1206 was pressed. Inaddition, the layout image displayed by resetting may be the layoutimage displayed by designating the double-page spread in the double-pagespread designation area 1202 (the layout image having been displayedbefore the template change process) or may be the layout image havingbeen displayed when the OK button 1205 was pressed. After that, the CPU101 proceeds again to S1105.

In S1111, the CPU 101 determines whether the press of the OK button 1205by the user is accepted. When making a YES determination, the CPU 101proceeds to S1112, and when making a NO determination, the CPU 101proceeds again to S1105.

In S1112, the CPU 101 specifies the layout image having been displayedat the time of press of the OK button 1205 as the layout image to beprinted corresponding to the double-page spread specified as the editingtarget double-page spread. Before the OK button 1205 is pressed, the CPU101 specifies the layout image generated by automatic layout before thetemplate change process or the layout image having been displayed at thetime of press of the OK button 1205 as the layout image to be printed.

After that, the CPU 101 terminates the process. Even after the press ofthe OK button 1205, the template change process may be executed.

The user can change and switch only the template for the layout image tobe printed, by the template change process as described above. After thetemplate change process, the user presses the put-into-cart button 1207to execute the album ordering process (output operation), therebyoutputting the layout image to be printed after the switching as analbum. In addition, the user can transmit the image data representingthe layout image to be printed after the switching to a device outsideof the image processing apparatus 100, for example.

As described above, in the present embodiment, when an operation forchanging the template for a layout image is accepted, the layout imagesusing the templates with a low degree of similarity to the template usedfor the pre-change layout image are suggested to the user.

Accordingly, the layout images using the templates possibly highlysatisfactory for the user can be suggested to the user. Eventually, theuser can select easily the appropriate (highly satisfactory) layoutimage as the post-change layout image.

Other Embodiments

In the foregoing embodiment, the layout image using a template with alow degree of similarity to the template used for the pre-change layoutimage is displayed as the post-change layout image. The method fordisplaying the post-change layout image is not limited to the foregoingone. For example, the layout image using the template with a low degreeof similarity to the template used for the pre-change layout image andanother layout image may be displayed to the user at the same time. Inaddition, at that time, the layout image using the template with a lowdegree of similarity to the template used for the pre-change layoutimage may be displayed in a more emphasized and prioritized manner thanthe other layout image. In addition, a plurality of layout images usingdifferent templates with a low degree of similarity to the template usedfor the pre-change layout image may be displayed. At that time, thelayout images using the template with a lower degree of similarity tothe template used for the pre-change layout image may be displayed in amore emphasized and prioritized manner than the other layout images. Asspecific examples for display in a more emphasized manner, apredetermined icon indicating a higher priority may be added to thelayout images using the template with a lower degree of similarity tothe template used for the pre-change layout image, or the frame line orcolor of the layout images may be made different from that of the otherlayout images.

For example, the layout image using a template with a high degree ofsimilarity to the template used for the pre-change layout image may bedisplayed as the post-change layout image. Whether to display the layoutimage using a template with a high degree of similarity to the templateused for the pre-change layout image on a priority basis or display thelayout image using a template with a low degree of similarity on apriority basis may be switchable by the user operation or the like.

In the foregoing embodiment, the template change process is executed onthe layout image generated by automatic layout. The present disclosureis not limited to this embodiment. For example, the template changeprocess of the present embodiment may be executed on the layout imagegenerated by the user manually selecting the images to be arranged inthe template or slots. That is, the layout image using the template witha low degree of similarity to the template selected by the user and thelayout image using the images selected by the user may be suggested tothe user on a priority basis.

In the foregoing embodiment, the template to be used for the post-changelayout image is specified by the degree of similarity to the templateused in the layout image corresponding to the editing target double-pagespread. However, the present disclosure is not limited to this mode butthe template to be used for the post-change layout image may bespecified by the degree of similarity to the template used for thelayout image corresponding to the double-page spread other than theediting target double-page spread. For example, the template to be usedfor the post-change layout image may be specified by the degree ofsimilarity to the template used for the layout image corresponding tothe double-page spreads (or either of them) before and after the editingtarget double-page spread. In addition, the template may be specified inconsideration taking into account both the degree of similarity to thetemplate used for the layout image corresponding to the editing targetdouble-page spread and the degree of similarity to the template used inthe layout image corresponding to the double-page spreads (or either ofthem) before and after the editing target double-page spread.

For example, when the template change process is performed on the layoutimage corresponding to a double-page spread other than the editingtarget double-page spread, the result of the template change process maybe referred to in the template change process on the layout imagecorresponding to the editing target double-page spread. For example, inthe template change process on the layout image corresponding to adouble-page spread other than the editing target double-page spread, atemplate with regular square slots tends to be selected as a template tobe used for the post-change layout image. In that case, referring to thetendency, the template corresponding to the tendency may be used in thetemplate change process on the layout image corresponding to the editingtarget double-page spread.

In the foregoing embodiment, when an instruction from the user isaccepted, the layout image using a template different from the templateused for the layout image corresponding to the editing targetdouble-page spread is displayed. However, the present disclosure is notlimited to this mode but the layout image using images different from(with a low degree of similarity to) the images used for the layoutimage corresponding to the editing target double-page spread may bedisplayed, for example. In this case, the same template may be usedbefore and after change. That is, in response to a user instruction, thelayout image different at least partly from the layout imagecorresponding to the editing target double-page spread and with a lowdegree of similarity to that layout image is preferably displayed. Onthe other hand, for example, when an instruction from the user isaccepted, a template that has no image arranged and is different fromthe template used for the layout image corresponding to the editingtarget double-page spread may be displayed.

The foregoing embodiment can also be implemented by executing thefollowing processes. That is, software (program) for carrying out thefunctions of the foregoing embodiment is supplied to a system or adevice via a network or any of various storage media, and a computer(CPU or MPU) in the system or the device reads and executes the program.The program may be executed on one computer or a plurality of computersin cooperation. All the processes described above may not be necessarilyimplemented by software but some or all of the processes may beimplemented by hardware such as ASIC. In addition, all the processes maynot be necessarily performed by one CPU but may be performed by aplurality of CPUs in cooperation as appropriate.

The present disclosure provides technological improvements orunconventional solutions in advancements and/or improvements in imageprocessing technology through use of specific manners of use withspecific limitations on available data and configurational states.

According to the present disclosure, it is easy for the user to specifya highly satisfactory layout image as a post-change layout image and ahighly satisfactory template as a template to be used for a post-changelayout image.

Embodiment(s) of the present disclosure can also be realized by acomputerized configuration(s) of a system or apparatus that reads outand executes computer executable instructions (e.g., one or moreprograms) recorded on a storage medium (which may also be referred tomore fully as a ‘non-transitory computer-readable storage medium’) toperform the functions of one or more of the above-describedembodiment(s) and/or that includes one or more circuits (e.g.,application specific integrated circuit (ASIC)) for performing thefunctions of one or more of the above-described embodiment(s), and by amethod performed by the computerized configuration(s) of the system orapparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computerized configuration(s) maycomprise one or more processors, one or more memories, circuitry, or acombination thereof (e.g., central processing unit (CPU), microprocessing unit (MPU), or the like), and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computerized configuration(s), for example, froma network or the storage medium. The storage medium may include, forexample, one or more of a hard disk, a random-access memory (RAM), aread only memory (ROM), a storage of distributed computing systems, anoptical disk (such as a compact disc (CD), digital versatile disc (DVD),or Blu-ray Disc (BD)™), a flash memory device, a memory card, and thelike.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of priority from Japanese PatentApplication No. 2017-193784, filed Oct. 3, 2017, which is herebyincorporated by reference herein in its entirety.

What is claimed is:
 1. A control method of an image processingapparatus, the method comprising: displaying on a display unit apre-change layout image in which an arrangement image is arranged in atemplate; accepting a change instruction for displaying a post-changelayout image different at least partly from the pre-change layout imageon the display unit; determining a degree of similarity each between thepost-change layout image that is each image included in a post-changelayout image group and the pre-change layout image; and executing, in acase that the change instruction is accepted, based on the determineddegree of similarity to the pre-change layout image, at least one of aprocess for displaying a first post-change layout image on the displayunit with a priority over a second post-change layout image included inthe post-change layout image group with a higher degree of similarity tothe pre-change layout image, determined in the determining, than thefirst post-change layout image included in the post-change layout imagegroup and a process for displaying the first post-change layout image onthe display unit in a more emphasized manner than the second post-changelayout image, wherein the degree of similarity in a predeterminedelement between the template used for the pre-change layout image andthe template used for the second post-change layout image is higher thanthe degree of similarity in the predetermined element between thetemplate used for the pre-change layout image and the template used forthe first post-change layout image, and wherein the post-change layoutimage with the lowest degree of similarity to the pre-change layoutimage is specified based on a table showing a value relating to thedegree of similarity in the predetermined element between the templateused for the pre-change layout image and a template identical to theformer template in one or more of elements other than the predeterminedelement.
 2. The control method according to claim 1, wherein thepost-change layout image is a layout image in which at least thearrangement image used for the pre-change layout image is arranged in atemplate different from the template used for the pre-change layoutimage.
 3. The control method according to claim 2, wherein a degree ofsimilarity between the template used for the pre-change layout image andthe template used for the second post-change layout image is higher thana degree of similarity between the template used for the pre-changelayout image and the template used for the first post-change layoutimage.
 4. The control method according to claim 1, further comprisingspecifying a post-change layout image with the lowest degree ofsimilarity to the pre-change layout image from the post-change layoutimage group, wherein in a case that the change instruction is accepted,at least one of a process for displaying the specified post-changelayout image specified as the post-change layout image with the lowestdegree of similarity to the pre-change layout image on the display unitwith a priority over the unspecified post-change layout image notspecified as the post-change layout image with the lowest degree ofsimilarity to the pre-change layout image and a process for displayingthe specified post-change layout image specified as the post-changelayout image with the lowest degree of similarity to the pre-changelayout image on the display unit in a more emphasized manner than theunspecified post-change layout image not specified as the post-changelayout image with the lowest degree of similarity to the pre-changelayout image is executed.
 5. The control method according to claim 1,wherein the predetermined element is one of an aspect ratio of a slotthat is a region included in a template where an image is arranged, ashape of the slot, an area of the slot, the number of the slots, acoordinate position of the slot in the template, the number of overlapsbetween the slots, and the number of coupling portions of the slots. 6.The control method according to claim 1, wherein, after at least theprocess for displaying the first post-change layout image on the displayunit with a priority over the second post-change layout image or theprocess for displaying the first post-change layout image on the displayunit in a more emphasized manner than the second post-change layoutimage is executed upon acceptance of the change instruction, when thechange instruction is accepted again, at least a process for displayinga third post-change layout image with a higher degree of similarity tothe pre-change layout image than the first post-change layout image onthe display unit with a priority over a fourth post-change layout imagewith a higher degree of similarity to the pre-change layout image thanthe third post-change layout image, or a process for displaying thethird post-change layout image on the display unit in a more emphasizedmanner than the fourth post-change layout image is executed.
 7. Thecontrol method according to claim 1, wherein, in a case that the changeinstruction is accepted, the pre-change layout image is no longerdisplayed on the display unit.
 8. The control method according to claim1, further comprising accepting, after at least the process fordisplaying the first post-change layout image on the display unit with apriority over the second post-change layout image or the process fordisplaying the first post-change layout image on the display unit in amore emphasized manner than the second post-change layout image isexecuted upon acceptance of the change instruction, a reset operation,wherein in a case that the reset operation is accepted, the pre-changelayout image is displayed on the display unit.
 9. The control methodaccording to claim 1, further comprising accepting, after at least theprocess for displaying the first post-change layout image on the displayunit with a priority over the second post-change layout image or theprocess for displaying the first post-change layout image on the displayunit in a more emphasized manner than the second post-change layoutimage is executed upon acceptance of the change instruction, an outputoperation, wherein in a case that the output operation is accepted, anoutput relating to the first post-change layout image is performed. 10.The control method according to claim 9, wherein the output relating tothe first post-change layout image is printing of the first post-changelayout image by an image forming apparatus or transmission of image datarepresenting the first post-change layout image to outside of the imageprocessing apparatus.
 11. The control method according to claim 1,wherein the process for displaying the first post-change layout image onthe display unit in a more emphasized manner than the second post-changelayout image is at least one of a process for adding a predeterminedicon to the first post-change layout image, a process for making a frameline of the first post-change layout image different from a frame lineof the second post-change layout image, and a process for making a colorof the first post-change layout image different from a color of thesecond post-change layout image.
 12. The control method according toclaim 1, wherein, in a case that the change instruction is accepted, thefirst post-change layout image is displayed on the display unit and thesecond post-change layout image is not displayed on the display unit toexecute the process for displaying the first post-change layout image onthe display unit with a priority over the second post-change layoutimage.
 13. The control method according to claim 1, further comprising:selecting automatically one or more template from a plurality oftemplates without accepting a selection instruction from the user; andselecting automatically one or more image data from a plurality of imagedata without accepting a selection instruction from the user, whereinthe pre-change layout image is an image in which an image represented bythe selected image data is arranged in the selected template.
 14. Thecontrol method according to claim 1, wherein the post-change layoutimage is a layout image in which at least an arrangement image differentfrom the arrangement image used for the pre-change layout image isarranged in the template used for the pre-change layout image.
 15. Thecontrol method according to claim 14, wherein the degree of similaritybetween the arrangement image used for the pre-change layout image andthe arrangement image used for the second post-change layout image ishigher than the degree of similarity between the arrangement image usedfor the pre-change layout image and the arrangement image used for thefirst post-change layout image.
 16. A control method of an imageprocessing apparatus comprising: displaying on a display unit apre-change layout image in which an arrangement image is arranged in atemplate; accepting a change instruction for displaying a post-changelayout image different at least partly from the pre-change layout imageon the display unit; determining a degree of similarity each between thepost-change layout image that is each image included in a post-changelayout image group and the pre-change layout image; and executing, in acase that the change instruction is accepted, based on the determineddegree of similarity to the pre-change layout image, at least one of aprocess for displaying a first post-change template on the display unitwith a priority over a second post-change template included in thepost-change layout image group with a higher degree of similarity to atemplate used for the pre-change layout image, determined in thedetermining, than the first post-change template included in thepost-change layout image group and a process for displaying the firstpost-change template on the display unit in a more emphasized mannerthan the second post-change template, wherein the degree of similarityin a predetermined element between the template used for the pre-changelayout image and the template used for the second post-change layoutimage is higher than the degree of similarity in the predeterminedelement between the template used for the pre-change layout image andthe template used for the first post-change layout image, and whereinthe post-change layout image with the lowest degree of similarity to thepre-change layout image is specified based on a table showing a valuerelating to the degree of similarity in the predetermined elementbetween the template used for the pre-change layout image and a templateidentical to the former template in one or more of elements other thanthe predetermined element.
 17. A control method of an image processingapparatus, the method comprising: displaying on a display unit apre-change layout image in which an arrangement image is arranged in atemplate; accepting a change instruction for displaying a post-changelayout image different at least partly from the pre-change layout imageon the display unit; acquiring a value relating to the degree ofsimilarity in the predetermined element between the template used forthe pre-change layout image and a template identical to the formertemplate in one or more of elements other than the predeterminedelement; determining a degree of similarity each between the post-changelayout image that is each image included in a post-change layout imagegroup and the pre-change layout image; and executing, in a case that thechange instruction is accepted, based on the determined degree ofsimilarity to the pre-change layout image, at least one of a process fordisplaying a first post-change layout image on the display unit with apriority over a second post-change layout image included in thepost-change layout image group with a higher degree of similarity to thepre-change layout image, determined in the determining, than the firstpost-change layout image included in the post-change layout image groupand a process for displaying the first post-change layout image on thedisplay unit in a more emphasized manner than the second post-changelayout image, wherein the value relating to the degree of similarity inthe predetermined element is indicated by a distance between a vectorbased on the predetermined element in the template used for thepre-change layout image and a vector based on the predetermined elementin the template identical to the former template in one or more of theelements other than the predetermined element, wherein a post changelayout image with the lowest degree of similarity to the pre-changelayout image is specified based on the acquired value, and wherein thedegree of similarity in a predetermined element between the templateused for the pre-change layout image and the template used for thesecond post-change layout image is higher than the degree of similarityin the predetermined element between the template used for thepre-change layout image and the template used for the first post-changelayout image.